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Abstract - In this paper, we analyze the results of a seven-month 
real-time streaming experiment, which was conducted between a 
number of unicast dialup clients, connecting to the internet through 
access points in more than 600 major U.S. cities, and a backbone 
video server. During the experiment, the clients streamed low- 
bitrate MPEG-4 video sequences from the server over paths with 
more than 5,000 distinct Internet routers. We describe the method- 
ology of the experiment, the architecture of our NACK-based 
streaming application, study end-to-end dynamics of 16 thousand 
ten-minute sessions (85 million packets), and analyze the behavior of 
the following network parameters: packet loss, round- trip delay, 
one-way delay jitter, packet reordering, and path asymmetry. We 
also study the impact of these parameters on the quality of real-time 
streaming. 

I. Introduction 

The Internet has become a complex interconnection of a 
large number of computer networks. The behavior of the 
Internet has been the target of numerous studies, but never- 
theless, the performance of the Internet from the perspective 
of an average home user still remains relatively undocu- 
mented. In addition, we believe that since end users are re- 
sponsible for a large fraction of Internet traffic, the study of 
network conditions experienced by these users is an impor- 
tant research topic. These are the two main reasons that com- 
pelled us to conduct a fundamentally different performance 
study that looks at Internet dynamics from the angle of an 
average Internet user. 

Even though the Internet has been extensively analyzed in 
the past, an overwhelming majority of previous studies were 
based on TCP traffic and involved high-speed bottleneck 
links. On the other hand, real-time streaming protocols have 
not received as much attention in these studies. Note that no 
previous work attempted to characterize the performance of 
real-time streaming in a large-scale Internet experiment of 
this sort (i.e., involving dialup access). The Internet has been 
studied from the perspective of TCP connections by Paxson 
[17], Bolliger et al. [4], Caceres et al [8], Mogul [15], and 
several others (e.g., [3]). Paxson's study included 35 geo- 
graphically distributed sites in 9 countries; Bolliger et al. 
employed 1 1 sites in 7 countries and compared the through- 
put performance of various implementations of TCP during a 
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six-month experiment; whereas the majority of other re- 
searchers monitored transit TCP traffic at a single backbone 
router [3], [15] or inside several campus networks [8] for the 
duration ranging from several hours to several days. 

The methodology used in both large-scale TCP experi- 
ments [4], [17] was similar and involved a topology where 
each participating site was paired with every other participat- 
ing site for an FTP-like transfer. Although this setup ap- 
proximates well the current use of TCP in the Internet, future 
entertainment-oriented streaming services, however, are more 
likely to involve a small number of backbone video servers 
and a large number of home users.' 

We believe that in order to study the current dynamics of 
real-time streaming in the Internet, we must take the same 
steps to connect to the Internet as an average end-user (i.e., 
through dialup ISPs). For example, ISPs often experience 
congestion in their own backbones, and during busy hours, 
V.90 (i.e., 56 kb/s) modems in certain access points are not 
available due to high user demand, none of which can be cap- 
tured by studying the Internet from a small campus network 
directly connected to the Internet backbone. 

In addition to choosing a different topological setup for the 
experiment, our work is different from the previous studies in 
several other aspects. First, the sending rate of a TCP connec- 
tion is driven by its congestion control, which can often cause 
increased packet loss and higher end-to-end delays in the path 
along which it operates (e.g., during slow start and congestion 
avoidance while probing for new bandwidth). In our experi- 
ment, we measured the end-to-end path dynamics as per- 
ceived by a CBR (constant bitrate) stream sharing backbone 
links with elastic TCP traffic. Hence, even though no active 
measurement study can avoid the effects of the Eisenberg 
principle, the use of CBR traffic allowed us to sample the real 
Internet performance without the bias of UDP-based conges- 
tion control applied to slow modem links* 

Second, TCP uses a positive ACK retransmission scheme, 
whereas real-time applications usually employ NACK-based 
retransmission to reduce the amount of traffic from the users 
to the streaming server. As a consequence, end-to-end path 
dynamics perceived by a NACK-based protocol could differ 


1 Our work focuses on non-interactive streaming applications where the user 
can. tolerate short (i.e., in the order of several seconds) startup delays (e.g., 
TV over the Internet). 

1 Without a doubt, future real-time streaming protocols will include some 
form of scalable congestion control; however, at the time of the experiment, 
it was not even clear which methods represented such congestion control. 
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from those sampled by TCP along the same path: real-time 
applications acquire samples of the round-trip delay (RTT) at 
rare intervals, send significantly less data along the path from 
the receiver to the sender, and bypass certain aspects of 
TCP's retransmission scheme (such as exponential timer 
backoff). Previous work [13] suggests that NACK-based re- 
transmission schemes may require a different retransmission 
timeout (RTO) estimator and leads us to believe that research 
in this area should be extended 

We should further mention that the Internet has been ex- 
tensively studied by various researchers using ICMP ping 
and traceroute packets [I], [9], [16], [17], [20], UDP echo 
packets [6], [7], and' multicast backbone (MBone) audio 
packets [24]. With the exception of the last one, similar ob- 
servations apply to these studies - neither the setup, nor the 
type of probe traffic represented realistic real-time streaming 
scenarios. In addition, among the studies that specifically sent 
video traffic over the Internet [5], [10], [21], [22], the major- 
ity of experiments involved only a few Internet paths, lasted 
for a very short period of time, and focused on analyzing the 
features of the proposed scheme rather than the impact of 
Internet conditions on real-time streaming. 

In this paper, we present the methodology and analyze the 
results of a seven-month large-scale real-time streaming ex- 
periment, which involved three nation-wide diaJup ISPs, each 
with several million active subscribers in the United States. 
The topology of the experiment consisted of a backbone 
video server streaming MPEG-4 video sequences to unicast 
home users located in more than 600 major U.S. cities. The 
streaming was performed in real-time (i.e., with a real-time 
decoder), utilized UDP for the transport of all messages, and 
relied on simple NACK-based retransmission to recover lost 
packets before their decoding deadlines. 

Even though we consider it novel and unique in many as- 
pects, there is an important limitation to our study. Our ex- 
periments document Internet path dynamics perceived by 
low-bitrate (i.e., modem-speed) streaming sessions. Recall 
that one of the goals of our work was to conduct a perform- 
ance study of the Internet from the angle of a typical home 
Internet user, and to this extent, we consider our work to be 
both thorough and successful. In addition, by focusing on 
low-bitrate paths, our study shows the performance of real- 
time protocols under the most difficult network conditions 
(i.e., large end-to-end delays, relatively high bit-error rates, 
low available bandwidth, etc.) and provides a "lower bound" 
on the performance of future Internet streaming applications, 

Despite this limitation, we believe that the results of our 
study conclusively establish the feasibility of video streaming 
in the currently best-effort Internet and provide a valuable 
insight into dynamics of real-time streaming from the per- 
spective of an average Internet user. 

The remainder of the paper is organized as follows. Section 
II describes the methodology of the experiment. Section III 
provides an overview of the experiment. Section IV studies 
packet loss, section V analyzes the frequency of underflow 
events, section VI looks at the behavior of the round-trip de- 


lay, section VII briefly discusses one-way delay jitter, section 
VIII examines the extent of packet reordering during the ex- 
periment, and section IX studies the phenomenon of asym- 
metric paths discovered during the experiment. Section X 
concludes the paper. 

II. METHODOLOGY 

A. Setup for the Experiment 

We started our work by attaching a Unix video server to 
the UUNET backbone via a Tl link (Figure I). To support 
the client's connectivity to the Internet, we selected three 
major nation-wide dialup ISPs (which we call ISP,, ISP^, and 
ISP,), each with at least five hundred V.90 (i.e., 56 kb/s) 
dialup numbers in the U.S., and designed an experiment in 
which hypothetical Internet users dialed a local access num- 
ber to reach the Internet (through one of our three ISPs) and 
streamed video sequences from the server. Although the cli- 
ents were physically placed in our lab in the state of New 
York, they dialed long-distance phone numbers and con- 
nected to the Internet through ISPs* access points located in 
each of the 50 states. Our database of phone numbers in- 
cluded 1813 different V.90 access points in 1 188 major U.S. 
cities. 



Figure 1. Topology of the experiment 

After the phone database was in place, we designed and 
implemented special software, which we call the dialer, that 
dialed phone numbers from the database, connected to the 
ISPs using the point-to-point protocol (PPP), issued aparallel 
traceroute to the server, and upon success, started the video 
client with the instructions to stream a ten-minute video se- 
quence from the server. Our implementation of traceroute 
(built into the dialer) used ICMP probes, sent all probes in 
parallel instead of sequentially (hence the name "parallel"), 
and recorded the IP time-to-live (TTL) 3 field of each returned 
'TTL expired" message. The use of ICMP packets and paral- 
lel traceroute facilitated much quicker discovery of routers, 
and the analysis of the TTL field in the returned packets al- 
lowed the dialer to compute the number of hops in the reverse 
path from each intermediate router to the client machine (us- 
ing a simple fact that each router reset the TTL field of each 
generated "TTL expired" packet to the value of the initial 


J Recall that the TTL field is decremented by 1 every time a packet is for- 
warded by a Ievel-3 (i.e., IP-level) device. 
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TTL ). Using the information about the number of forward 
and reverse hops for each router, the dialer was able to detect 
asymmetric end-to-end paths, which we study in section VIII. 

In our analysis of the data, we attempted to isolate clearly 
modem-related pathologies (such as packet loss caused by a 
poor connection over the modem link and large RTTs due to 
data-link retransmission) from those caused by congested 
routers of the Internet Thus, connections that were unable to 
complete a traceroute to the server, connections with high bit- 
error rates (BER), and connections during which the modem 
could not sustain our streaming rates were all considered use- 
less for our study and were excluded from the analysis in this 
paper. 

In practice, to avoid studying connections with clearly in- 
sufficient end-to-end bandwidth and various modem-related 
problems, we utilized the following methodology. We de- 
fined a streaming attempt through a particular access number 
to be successful, if the ISP's access number was able to sus- 
tain the transmission of our video stream for its entire length 
at the stream's target IP bitrate r. To be specific, the video 
client terminated connections in which the aggregate (i.e., 
counting from the very beginning of a session) packet loss 
grew beyond a certain threshold p p or the aggregate incom- 
ing bitrate dropped below another threshold p r . The experi- 
ments reported in this paper used $ p equal to 15% and |3 r 
equal to 0.9r, both of which were experimentally found to be 
necessary conditions for efficient filtering out of modem- 
related failures. The packet-loss threshold was activated after 
1 minute of streaming and the bitrate threshold after 2 min- 
utes to make sure that slight fluctuations in packet loss and 
incoming bitrate at the beginning of a session were not mis- 
taken for poor connection quality. After a session was over, 
the success or failure of the session was communicated from 
the video client to the dialer, the latter of which kept track of 
the time of day and the phone number that either passed or . 
failed the streaming test 

In order to make the experiment reasonably short, we con- 
sidered all phone numbers from the same state to be equiva- 
lent, and consequently, we assumed that a successful stream- 
ing attempt through any phone number of a state indicated a 
successful coverage of the state regardless of which phone 
number was used. Furthermore, we divided each 7-day week 
into 56 three-hour timeslots (i.e., 8 timeslots per day) and 
designed the dialer to select phone numbers from the data- 
base in such order so that each state would be successfully 
covered within each of the 56 timeslots at least once. In other 
words, each ISP needed to sustain exactly 50.56 = 2,800 suc- 
cessful sessions before the experiment was allowed to end. 

B. Real-time Streaming 

For the purpose of the experiment, we used an MPEG-4 
encoder to create two ten-minute QCIF (176x144) video 


4 The majority of routers used the initial TTL equal to 255, while some ini- 
tialized the field to 30, 64, or 128. Subtracting the received TTL from the 
initial TTL produced the number of hops along the reverse path. 


streams. The first stream, which we call S\, was coded at the 
video bitrate of 14 kb/s (size 1.05 MBytes), and the second 
steam, which we call S 2 » was coded at 25 kbfs (size 1.87 
MBytes). The experiment with stream S\ lasted during No- 
vember - December 1 999 and the one with stream 5 2 was an 
immediate follow-up during January - May 2000. 

During the transmission of each video stream, the server 
split it into 576-byte IP packets. Stream S\ consisted of 4,188 
packets, and stream 5 2 consisted of 5,016 packets. Video 
frames always started on a packet boundary, and conse- 
quently, the last packet in each frame was allowed to be 
smaller than others (in fact, many P (prediction-coded) 
frames were smaller than the maximum payload size and 
were carried in a single UDP packet). As a consequence of 
packetization overhead, the IP titrates (i.e., including IP, 
UDP, and our special 8-byte headers) for streams S\ and S 2 
were 16.0 and 27.4 kb/s, respectively. 

In our streaming experiment, the term real-time refers to 
the fact that the video decoder was running in real-time. Re- 
call that each compressed video frame has a specific decod- 
ing deadline, which is usually based on the time of the 
frame's encoding. If a compressed video frame is not fully 
received by the decoder buffer at the time of its deadline, the 
video frame is discarded and an underflow event is registered. 
Moreover, to simplify the analysis of the results, we imple- 
mented a strict real-time decoder model, in which the play- 
back of the arriving frames continued at the encoder-specified 
deadlines regardless of the number of underflow events (i.e., 
the decoding deadlines were not adjusted based on network 
conditions). 

In addition, many CBR (constant bitrate) video coding 
schemes include the notion of the ideal startup delay /18], 
[ 19] (the delay is called "ideal" because it assumes a network 
with no packet loss and a constant end-to-end delay). This 
ideal delay must always be applied to the decoder buffer be- 
fore the decoding process may begin. The ideal startup delay 
is independent of the network conditions and solely depends 
on the decisions made by the encoder during the encoding 
process. 5 On top of this ideal startup delay, the client in a 
streaming session usually must apply an additional startup 
delay in order to compensate for delay jitter (i.e., variation in 
the one-way delay) and permit the recovery of lost packets 
via retransmission. This additional startup delay is called the 
delay budget {D budge ^ and reflects the values of the expected 
(at the beginning of a session) delay jitter and round-trip de- 
lay during the length of the session. Note that in the context 
of Internet streaming, it is common to call D^ get simply 
"startup delay'* and to completely ignore the ideal startup 
delay (e.g., [lo]). From this point on, we will use the same 
convention. In all our experiments, we used D Wjf/ equal to 
2,700 ms, which was manually selected based on preliminary 
testing. Consequently, the total startup delay (observed by an 


We will not elaborate further on the ideal startup delay, except mention that 
it was approximately 1,300 ms for each stream. 
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end-user) at the beginning of each session was approximately 
4 seconds. 

C. CI ten t-Server A rch itecture 

For the purpose of our experiment, we implemented a cli- 
ent-server architecture for MPEG-4 streaming over the Inter- 
net. The server was fully multithreaded to ensure that the 
transmission of packetized video was performed at the target 
IP bitrate of each streaming session and to provide quick re- 
sponse to clients' NACK requests. The streaming was im- 
plemented in bursts of packets (with the burst duration D A 
varying between 340 and 500 ms depending on the bitrate) 
for the purposes of making the server as low-overhead as 
possible (for example, RealAudio servers use D b = 1,800 ms 
[14]). Although we agree that in many cases the desired way 
of sending constant bitrate (CBR) traffic is to equally space 
packets during transmission, there are practical limitations 
(such as OS scheduling and inter-process switching delays) 
that often do not allow us to follow this model. 

The second and the more involved part of our architecture, 
the client, was designed to recover lost packets through 
NACK-based retransmission and collect extensive statistics 
about each received packet and each decoded frame. Fur- 
thermore, as it is often done in NACK-based protocols, the 
client was in charge of collecting round-trip delay (RTT) 
samples (the resolution of timestamps was 100 microsec- 
onds). The measurement of the RTT involved the following 
two methods. In the first method, each successfully recovered 
packet provided a sample of the RTT (i.e., the RTT was the 
duration between sending a NACK and receiving the corre- 
sponding retransmission). In our experiment, in order to 
avoid the ambiguity of which retransmission of the same 
packet actually returned to the client, the header of each 
NACK request and each retransmitted packet contained an 
extra field specifying the retransmission number of the 
packet. 

The second method of measuring the RTT was used by the 
client to obtain additional samples of the round-trip delay in 
cases when network packet loss was too low. The method 
involved periodically sending simulated retransmission re- 
quests to the server if packet loss was below a certain thresh- 
old. In response to these simulated NACKs, the server in- 
cluded the usual overhead 6 of fetching the needed packets 
from the storage and sending them to the client. In our ex- 
periment, the client activated simulated NACKs, spaced 30 
seconds apart, if packet loss was below 1%. 

We tested the software and the concept of a wide-scale ex- 
periment of this sort for nine months before we felt comfort- 
able with the setup, the reliability of the software, and the 
exhaustiveness of the "collected statistics. In addition to exten- 
sive testing of the prototype for nine months, we monitored 
various statistics reported by the clients in real-time (i.e., on 


6 The server overhead was below 10 ms for all retransmitted packets and did 
not have a major impact on our characterization of the RTT process later in 
this paper. 


the screen) during the experiments for sanity and consistency 
with previous tests. 

Our traces consist of six datasets, each collected by a dif- 
ferent machine. Throughout this paper, we will use notation 
D n x to refer to the dataset collected by the client assigned to 
ISPjf (x = a, b t c) during the experiment with stream S n (n = 1 , 
2). Furthermore, we will use notation to refer to the com- 
bined set {D n a uD n b uD tt c }. 

III. Experiment Overview 

In dataset D u the three clients performed 16,783 long- 
distance connections to the ISPs' remote modems and suc- 
cessfully completed 8,429 streaming sessions. 7 In D 2 , the 
clients performed 17,465 modem connections and sustained 
8,423 successful sessions. Analysis of the above numbers 
suggests that in order to receive real-time streaming material 
with a minimum quality at 16 to 27.4 kb/s, an average U.S. 
end-user, equipped with a V.90 modem, needs to make ap- 
proximately two dialing attempts to the ISPs' phone numbers 
within the state where the user resides. The success rate of 
streaming sessions during different times of the day is illus- 
trated in Figure 2. Note the dip by a fector of two between the 
best and the worst times of the day. 

Furthermore, in dataset D h the clients traced the arrival of 
37.7 million packets, and in D2, the arrival of additional 47.3 
million (for a total of 85 million). In terms of bytes, the first 
experiment transported 9.4 GBytes of video data and the sec- 
ond one transported another 17.7 GBytes (for a total of 27.1 
GBytes). 
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Figure 2. Success of streaming attempts during the day. 

Recall that each experiment lasted as long as it was needed 
to cover the entire United States. Depending on the success 
rate within each state, the access points used in the experi- 
ment comprised a subset of our database. In D u the experi- 
ment covered 962 dialup points in 637 U.S. cities, and inDi, 
it covered 880 dialup points in 575 U.S. cities. Figure 3 
shows the combined (i.e., including both datasets Diand D 2 ) 
number of distinct cities in each state covered by our experi- 
ment (1003 access points in 653 cities). 


7 Typical reasons for failing a session were PPP-layer connection problems, 
inability to reach the server (i.e., failed traceroute), high bit-error rates, low 
(14.4-19.2 kh/s) connection rates, and insufficient bandwidth. 
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During the experiment, each session was preceded by a 
parallel traceroute, which recorded the IP addresses of all 
discovered routers (DNS and WH'OtS* lookups were done 
off-line after the experiments were over). The average time 
needed to trace an end-to-end path was 1,731 ms, 90% of the 
paths were traced under 2.5 seconds, and 98% under 5 sec- 
onds. Dataset D\ recorded 3,822 distinct Internet routers, D 7 
recorded 4,449 distinct routers, and both experiments com- 
bined produced the IP addresses of 5,266 unique muters. The 
majority of the discovered routers belonged to the ISPs' net- 
works (5 1%) and UUNET (45%), which confirmed our intui- 
tion before the experiment that all three ISPs had numerous 
(i.e., hundreds of) direct peering connections with UUNET 
(some of this information could also have been extracted 
from BGP tables). Moreover, our traces recorded approxi- 
mately 200 muters that belonged to five additional Autono- 
mous Systems (AS). 



Figure 3. The number of cities per state that participated in cither D t or Z>j. 

The average end-to-end hop count was 1 1.3 in D\ ^mini- 
mum and 17 maximum) and 1 ] .9 in (6 minimum and 22 
maximum). Figure 4 shows the distribution of the number of 
hops in the encountered end-to-end paths in each of £), and 
D 2 . As the figure shows, the majority of paths (75% inDj and 
65% in Di) contained between 10 and 13 hops. 



Figure 4. Distribution of the number of end-to^ntf hops 


Throughout the rest of the paper, we restrict ourselves to 
studying only successful (as defined in section II. A) sessions 
in both D\ and D 2 ; We call these new purged datasets (with 
only successful sessions) D [p and respectively (purged 
datasets are defined similarly for n = 1, 2 and;n = a, b, c). 


s The WHOIS database was used to discover the; Autonomous System (AS) 
of each router. 


Recall that {D^uD^} contains 16,852 successful sessions, 
which are responsible for 90% of the bytes and packets, 73% 
of the routers, and 74% of the U.S. cities recorded in 

IV. Packet Loss 

A. Overview 

Numerous researchers have studied Internet packet loss, 
and due to the enormous diversity of the Internet, only few 
studies agree on the average packet loss rate and the average 
loss burst length (i.e., the number of packets lost in a row). 
Among numerous studies, the average Internet packet loss 
was reported to vary between 1 1% and 23% by Bolbt [6] de- 
pending on the inter-transmission spacing between packets, 
between 0.36% and 3.54% by Borellae/ al.[7] depending on 
the studied path, between 1.38% and 1 1% by Yajnik et ai 
[24] depending on the location of the MBone receiver, and 
between 2.7% and 5.2% by Paxson [17] depending on the 
year of the experiment. In addition, 0.49% average packet 
loss rate was recently reported by Balakrishnan et al. [3], who 
analyzed the dynamics of a large number of TCP web ses- 
sions at a busy Internet server. 

In dataset D Xp , the average recorded packet loss rate was 
0.53% and in D 7p> it was 0.58%. Even though these rates are 
much lower* than those traditionally reported by Internet re- 
searchers during the last decade, they are still somewhat 
higher than those reported by backbone ISPs [23]. Further- 
more, 38% of the sessions in {D\ p \jD2 P } did not experience 
any packet loss, 75% experienced loss rates below 0.3%, and 
91% experienced loss rates below 2%. On the other hand, 2% 
of the sessions suffered packet loss rates 6% or higher. 

In addition, as we expected, average packet loss rates ex- 
hibited a wide variation during the day. Figure 5 shows the 
evolution of loss rates as a function of the timeslot (i.e., the 
time of day), where each point represents the average of ap- 
proximately 1,000 sessions. As the figure shows, the varia- 
tion in loss fates between the best (3-6 am) and die worst (3-6 
pm) times of the day was almost by a factor of four. 
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Figure 5. Average packet loss rates during the day. 


* Note that during the experiment, simply dialing a different access number 
in most cases fixed the problem of high packet loss. This Fact sho ws . that the 
majority of fatted sessions documented pathologies created by the. modem 
(or the access point) rather than the actual .packet loss in the Internet. Since 
an end-user can always re-dial a bad connection searching for better network 
conditions, we believe that the bias created by removing failed sessions 
reflects the. actions of a typical Internet user. 
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The apparent discontinuity in Figure 5 between timeslots 7 
(21:00-0:00) and 0 (0:00-3:00) is due to the coarse timescale 
used in the figure. On finer timescales (e.g., minutes), loss 
rates converge to a common value near midnight. A similar 
discontinuity in packet loss rates was reported by Paxson [ 17] 
for North American sites, where packet loss during timeslot 7 
was approximately twice as high as that during timeslot 0. 

The variation in the average per-state packet loss (as 
shown in Figure 6) was quite substantial (from 0.2% in Idaho 
to 1 .4% in Oklahoma), but virtually did not depend on the 
state's average number of end-to-end hops (correla- 
tion coefficient p was -0.04) or the state's average RTT (cor- 
relation -0.16). However, as we will see later, the average 
per-state RTT and the number of end-to-end hops were in feet 
positively correlated. 



Figure 6. Average per state packet loss rates. 

B. Loss Burst Lengths 

We next attempt to answer the question of how bursty 
Internet packet loss was during the experiment. Figure 7 
shows the distribution (both the PDF and the CDF) of loss 
burst lengths in {D\ p uD-^} (without loss of generality, the 
figure stops at burst length 20, covering more than 99% of the 
bursts). Even though the upper tail of the distribution had 
very few samples, it was fairly long and reached burst lengths 
of over 100 packets. 
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Figure 7. PDF and CDF functions of loss burst lengths in {D lp vDip}. 

Figure 7 is based on 207,384 loss bursts and 431,501 lost 
packets. The prevalence of single packet losses, given the feet 
that packets in our experiment were injected into the Internet 


in bursts at the Tl speed", leads us to speculate that either 
router queues sampled .in our experiment overflowed on time- 
scales smaller than the time needed to transmit a single IP 
packet over a Tl link (i.e., 3 ms for the largest packets and 
1.3 ms for the average-size packets), or that backbone routers 
employed Random Early Detection (RED) for preventing 
congestion. 

To investigate the presence of RED in the Internet, we con- 
tacted several backbone and dialup ISPs whose routers were 
recorded in our trace data and asked them to comment on the 
deployment of RED in their backbones. Among the ISPs that 
responded to our request, the majority had purposely disabled 
RED and the rest were running RED only for select custom- 
ers at border routers, but not on the public backbone. 

Ruling out RED, a second look at the situation reveals that 
server's possible interleaving of packets from different ses- 
sions could have expanded the inter-packet transmission dis- 
tance of each flow by up to a factor of 3. Furthermore, before 
each lost packet reached the corresponding congested router, 
packets from other Internet flows could have queued immedi- 
ately behind the lost packet, effectively expanding the inter- 
packet distance even further. Therefore, even though our data 
point toward transient buffer overflow events during the ex- 
periment (i.e., 1-3 ms), this conclusion may not necessarily 
hold in all cases. 

Furthermore, as previously pointed out by many research- 
ers (e.g., [24]), the upper tail of loss burst lengths usually 
contains a substantial percentage of all lost packets. In each 
of Dip and D 2p , single-packet bursts contained only 36% of all 
lost packets, bursts two packets or shorter contained 49%, 
bursts 10 packets or shorter contained 68%, and bursts 30 
packets or shorter contained 82%. At the same time, 13% of 
all lost packets were dropped in bursts at least 50 packets 


Traditionally, the burstiness of packet loss is measured by 
the average loss burst length. In the first dataset (D ]p ) t the 
average burst length was 2.04 packets. In the second dataset 
(Z)^), the average burst length was slightly higher (2.10), but 
not high enough to conclude that the higher bitrate of stream 
S 2 was clearly responsible for burstier packet loss. Further- 
more, the conditional probability of packet loss, given that 
the previous packet was also lost, was 5 1% in D\ p and 53% in 
D 2p . These numbers are consistent with those previously re- 
ported in the literature. Bolot [6] observed the conditional 
probability of packet loss to range from 18% to 60% depend- 
ing on inter-packet spacing during transmission, Borella et al. 
[7] from 10% to 35% depending on the time of day, and Pax- 
son [17] reported 50% conditional probability for loaded (i.e., 
queued behind the previous) TCP packets and 25% for 
unloaded packets. Using Paxson's terminology, the majority 


10 The server was only involved in low-bitratc streaming for our clients and 
did not have a problem blasting bursts of packets at the full speed of the 
adjacent link (Le., 10 mb/s). The spacing between packets was further ex- 
panded by the Tl link to UUNET. 
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of our packets were loaded since the server sent packets in 
bursts at a rate higher than the bottleneck link's capacity. 

C. Loss Burst Durations 

To a large degree, the average loss burst length depends on 
how closely the packets are spaced during transmission. As- 
suming that bursty packet loss comes from buffer overflow 
events in drop-tail queues rather than from consecutive hits 
by RED or from bit-level corruption, it is clear that all pack- 
ets of a flow passing through an overflown router queue will 
be dropped for the duration of the instantaneous congestion. 
Hence, the closer together the flow's packets arrive to the 
router, the more packets will be dropped during each queue 
overflow. This fact was clearly demonstrated in Bolot's ex- 
periments [6], where UDP packets spaced 8 ms apart suffered 
larger loss burst lengths (mean 2.5 packets) than packets 
spaced 500 ms apart (mean 1.1 packets). Yajnik et al. [24] 
reported a similar correlation between loss burst lengths and 
the distance between packets. Consequently, instead of ana- 
lyzing burst lengths, one might consider analyzing burst dura- 
tions since the latter does not depend on inter-packet spacing 
during transmission. 

Using our traces, we can only infer an approximate dura- 
tion of each loss burst, because we do not know the exact 
time when the lost packets were supposed to arrive to the 
client (i.e., knowing the exact arrival time would require the 
knowledge of delay jitter experienced by the lost packet; 
however, since they were lost, this information is unavail- 
able). Hence, for each loss event, we define the loss burst 
duration as the time elapsed between the receipt of the packet 
immediately preceding the loss burst and the packet immedi- 
ately following the loss burst. Figure 8 shows the distribution 
(CDF) of loss burst durations in seconds. Although the distri- 
bution tail is quite long (up to 36 seconds), the majority 
(more than 98%) of loss burst durations in both datasets D Xp 
and Dty fall under 1 second. Paxson's study [ 17] also ob- 
served large loss burst durations (up to 50 seconds), however, 
only 60% of the loss bursts studied by Paxson were contained 
below 1 second. In addition, our traces showed that the aver- 
age distance between lost packets in the experiment was 1 72- 
188 good packets, or 2 1-27 seconds, depending on the 
streaming rate. 
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Figure 8. The CDF function of loss burst durations in {D\ p \jD2p\ 


D. Heavy Tails 

In conclusion of this section, it is important to note that 
packet losses sometimes cannot be modeled as independent 
events due to buffer overflows that last long enough to affect 
multiple adjacent packets. Consequently, future real-time 
protocols should expect to deal with bursty packet losses 
(Figure 7) and possibly heavy-tailed distributions of loss 
burst lengths (see below). 

Several researchers reported a heavy-tailed nature of loss 
burst lengths, and the shape parameter a of the Pareto distri- 
bution fitted to the length (or duration) of loss bursts was 
recorded to range from 1 .06 (Paxson [ 1 7]) to 2.75 (Borella et 
al. [7]). On the other hand, Yajnik et al. [24] partitioned the 
collected data into stationary segments and reported that 1 loss 
burst lengths could be modeled as exponential (i.e., not 
heavy-tailed) within each stationary segment. In addition, 
Zhang et al. [25] reported that packet loss along some Inter- 
net paths was stationary and could be modeled as exponen- 
tial, whereas other paths were found to be non-stationary and 
not easy to model. 

Using intuition, it is clear that packet loss and RTT random 
processes in both D\ p and Dip are expected to be non- 
stationary. For example, the non-stationarity can be attributed 
to the time of day or the location of the client. In either case, 
we see three approaches to modeling such non-stationary 
data. In the first approach, we would have to analyze 16,852 
PDF functions (one for each session) for stationarity and 
heavy tails. Unfortunately, an average session contained only 
24 loss bursts, which is insufficient to build a good distribu- 
tion function for a statistical analysis. 

The second approach would be to combine all sessions into 
groups, which are intuitively perceived to be stationary (e.g., 
according to the access point or the timeslot), and then per- 
form similar tests for stationarity and heavy tails within each 
group. We might consider this direction for future work as it 
appears to be the right way of modeling random variables 
sampled over diverse Internet paths and during different 
times of the day. 

The third approach is to do what the majority has done in 
the past assume that all data samples belong to a stationary 
process and are drawn from a single distribution. Using this 
last approach, Figure 9 shows a log-log plot of the comple- 
mentary CDF function from Figure 7 with a least-squares tit 
of a straight line representing, a hypeibolic (i.e., heavy-tailed) 
distribution (the dotted curve is the exponential distribution 
fitted to the data). The tit of a straight line is quite good (with 
correlation p = 0,99) and provides a strong indication that the 
distribution of loss burst lengths in the combined dataset 
{D ]p uD2p} is heavy-tailed. Furthermore, as expected, we 
notice that the exponential distribution in Figure 9 decays too 
quickly to even remotely fit the data. 

Finally, consider a Pareto distribution with a CDF F(x) ~ 
Hp/x) a and PDF y(;t) = apV \ where a is the shape pa- 
rameter and P is the location parameter. Using Figure 9, we 
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establish that a Pareto distribution with a = 1.34 (finite mean, 
but infinite variance) and p = 0.65 fits our data very well. 
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Figure 9. The complimentary CDF of loss burst lengths in {D^vD^} on a 
log-log scale fitted with hyperbolic (straight line) and exponential (dotted 
curve) distributions. 

V. Underflow Events 

The impact of packet losses on real-time applications is 
understood feirly well. Note that in this paper, we use a fairiy 
simplified (but nevertheless very realistic) assumption that 
each video frame with a least one lost packet causes an un- 
derflow event. There are several reasons for doing so. First, 
most compression protocols (including MPEG-4) do not 
standardize error concealment techniques that are aimed at 
concealing the loss of an entire packet (i.e., at most, they re- 
cover from bit errors). Packet loss concealment techniques 
vary from obvious to extremely complex, and we tried to 
avoid turning this paper into an evaluation study of any par- 
ticular concealment technique. Second, the performance of 
error concealment varies greatly depending on the amount of 
motion in the video, the location of the lost packet (i.e., I, P, 
or B frame), and the compression method used. Furthermore, 
in many cases, concealing a loss of an entire packet is simply 
impossible (i.e., when the bitrate is low, and especially cases 
when the entire frame fits into a single packet). Conse- 
quently, this paper does not consider error concealment in 
conjunction with packet loss, and each video frame with 
missing packets at the time of decoding results in an under- 
flow event. 

In addition, this paper does not discuss any video quality 
related metrics (e.g., peak signal to noise ratio (PSNR)) of the 
received stream. Even though the user is ultimately interested 
in the video quality of the stream, the video performance de- 
pends on many different parameters (i.e., encoder options, 
GOP structure, the amount of motion in the video, etc.) and 
provides little information about the actual network condi- 
tions in the Internet 

We should further mention that all groups of pictures 
(GOPs) in both video streams were composed of I and P 
fames, which means that a loss of a single frame propagated 
until the end of the corresponding GOP (i.e., a single under- 
flow event typically caused several other underflow evens in 
the same GOP). Note that somewhat better performance can 


be achieved if the video stream is coded with B frames; how- 
ever, the analysis of such streams is impossible without con- 
ducting a brand new experiment (i.e., it is beyond the scope 
of this paper). 

In addition to packet loss, real-time applications suffer 
from large end-to-end delays. However, not all types of delay 
are equally important to real-time applications. As we will 
show below, one-way delay jitter was responsible for 90 
times more underflow events in our experiment than packet 
loss combined with large RTTs. 

Delays are important for two reasons. First, large round- 
trip delays make retransmissions late for their decoding dead- 
lines. However, the RTT is important only to the extent of 
recovering lost packets and, in the worst case, can cause only 
lost packets to be late for decoding. On the other hand, the 
second kind of delay, delay jitter (i.e., one-way delay varia- 
tion), can potentially cause each data (Le., non-retransmitted) 
packet to be late for decoding. 

Consider the following. In {D^KjDy,}, packet loss affected 
431,501 packets, out of which 159,713 (37%) were discov- 
ered to be missing after their decoding deadlines had passed, 
and consequently, NACKs were not sent for these packets. 
Out of 271,788 remaining lost packets, 257,065 (94.6%) were 
recovered before their deadlines, 9,013 (3.3%) arrived late, 
and 5,710 (2.1%) were never recovered. The fact that more 
than 94% of "recoverable" lost packets were actually re- 
ceived before their deadlines indicates that retransmission is a 
very efficient method of overcoming packet loss in real-time 
applications. Clearly, the success rate will be even higher in 
networks with smaller end-to-end delays. 

Before we study underflow events caused by delay jitter, 
let us introduce two types of lute retransmissions. The first 
type consists of packets that arrived after the decoding dead- 
line of the last frame of the corresponding group of pictures 
(GOP). These packets were completely useless and were dis- 
carded. The second type of late packets, which we call par- 
tially late, consists of those packets that missed their own 
decoding deadline, but arrived before the deadline of the last 
frame of the same GOP. Since the video decoder in our ex- 
periment could decompress frames at a substantially higher 
rate than the target fps, the client was able to use partially late 
packets for motion-compensated reconstruction of the re- 
maining frames from the same GOP before their correspond- 
ing decoding deadlines (i.e., partially late packets were used 
to "rescue" the remaining frames from the same GOP). Out 
of 9,013 late retransmissions, 4042 (49%) were partially late. 
Using each partially late packet, the client was able to "res- 
cue" on average 4.98 frames from the same GOP" inD Jp and 
4.89 frames in by employing the above-described catch- 
up decoding technique (for more discussion, see [20]). 

The second type of delay, one-way delay jitter, caused 
1,167,979 data (i.e., non-retransmitted) packets to miss their 
decoding deadlines. Hence, the total number of under-low 
(i.e., missing at the time of decoding) packets was 159,713 + 


11 We used lO- frame GOPs in both sequences. 
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9,0i3 + 5,710 + 1,167,979 = 1,342,415 (1.7% of the number 
of sent packets), which means that 98.9% of underflow pack- 
ets were created by large one-way delay jitter, rather than by 
pure packet loss. Even if the clients had not attempted to re- 
cover any lost packets, still 73% of the missing packets at the 
time of decoding would have been caused by large delay jit- 
ter. Furthermore, these 1.3 million underflow packets caused 
a "freeze-frame" effect for the average duration of 10.5 sec- 
onds per ten-minute session in D\ p and 8.6 seconds in D^, 
which can be considered excellent given the small amount of 
delay budget (i.e., startup delay) used in the experiments. 

To further understand the phenomenon of late packets, we 
plotted in Figure 10 the CDFs of the amount of time by which 
late packets missed their deadlines (i.e., the amount of time 
that we need to add to delay budget D bu<fg€t = 2,700 ms in or- 
der to avoid a certain percentage of underflow events) for 
both late retransmissions and late data packets. As the figure 
shows, 25% of late retransmissions missed their deadlines by 
more than 2.6 seconds, 10% by more than 5 seconds, and 1% 
by more than 10 seconds (the tail of the CDF extends up to 
98 seconds). At the same time, one-way delay jitter had a 
more adverse impact on data packets - 25% of late data pack- 
ets missed their deadlines by more than 7 seconds, 10% by 
more than 13 seconds, and 1% by more than 27 seconds (the 
CDF tail extends up to 56 seconds). 

The only way to reduce the number of late packets caused 
by both large RTTs and delay jitter is to apply a larger startup 
delay Dy udset at the beginning of a session (in addition to 
freezing the display and adding extra startup delays during 
the session, which was not acceptable in our model). Hence, 
for example, Internet applications utilizing a 13-second delay 
budget (which corresponds to 10.3 seconds of additional de- 
lay in Figure 10) would be able to recover 99% of retransmis- 
sions and 84% of data packets that were late during the ex- 
periment, given similar streaming conditions. 
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Figure 10. CDF functions of the amount of time by which retransmitted and 
data packets were late for decoding. 

VI. RouNd-trip Delay 

A. Overview 

Figure 11 shows the PDF functions of the round-trip delay 
in each of D lp and (660,439 RTT samples in both data- 
sets). Although the tail of the combined distribution reached 


the enormous values of 126 seconds for simulated and 102 
seconds for real retransmissions, the majority (75%) of the 
samples were below 600 ms, 90% below 1 second, and 
99.5% below 10 seconds. The average RTT was 698 ms in 
D {p and 839 ms in Dy. The minimum RTT was 119 and 172 
ms, respectively. Although very rare, extremely high RTTs 
were found in all six datasets Ap° - Dty (as a reminder, data- 
set D np x consists of successful sessions involving stream S n 
and recorded through ISP X ). Furthermore, out of more than 
660 thousand RTT samples in {D^vD^}, 437v/eie at least 
30 seconds, 32 at least 50 seconds, and 20 at least 75 seconds. 
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Figure 11. PDF functions of the RTT samples in each of D\ p and Dip. 

Although pathologically high RTTs may seem puzzling at 
first, there is a simple explanation. Modem error correction 
protocols (i.e., the commonly used V.42) implement retrans- 
mission for corrupted blocks of data on the physical layer. 12 
Error correction is often necessary if modems negotiated data 
compression (i.e., V.42bis) over the link and is desirable if 
PPP Compression Control Protocol (CCP) is enabled on the 
data-link layer. In all our experiments, both types of com- 
pression were enabled, imitating the typical setup of a home 
user. Therefore, if a client established a connection to a re- 
mote modem at a low bitrate (which was sometimes accom- 
panied by a significant amount of noise in the phone line), 
each retransmission on the physical layer took a large time to 
complete before the data was delivered to the upper layers. In 
addition, large IP-level buffers on either side of the modem 
link fhrther delayed packets arriving to or originating from 
the client host. 

Note that the purpose of classifying sessions into foiled and 
successful in section ILA was to avoid reporting pathological 
conditions caused by the modem links. Since only a handful 
(less than 0.5%) of RTTs in {D^uD^} were seriously ef- 
fected by modem-level retransmission and bit errors (we con- 
sider sessions with RTTs higher than 10 seconds to be caused 
by modem-related problems 1 ), we conclude that our heuristic 


'* Since the telephone network beyond the local loop in the U.S. is mostly 
digital, we believe that dialing long-distance (rather than local) numbers had 
no significant effect on the number of bit errors during the experiment. 
u For example, one of the authors uses a modem access point at home with 
IP-level buffering on the ISP side equivalent to 6.7 seconds. Consequently, 
delays as high as 5-10 seconds may often be caused by non-pathological 
conditions. 
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was successful in filtering out the majority of pathological 
connections and that future application-layer protocols, run- 
ning over a modem link, should be prepared to experience 
RTTs in the onler of several seconds. 

Furthermore, the removal of sessions with RTTs higher 
than 10 seconds does not change any of the results below and, 
at the same time, prohibits us from showing the extent of 
variation in the network parameters experienced by a home 
Internet user. Therefore, since all sessions in {D\ p \jD 2p } woe 
able to successfully complete, we consider the removal of 
sessions based on their large RTT to be unwarranted. 

B, Heavy Tails 

Mukherjee [ 16J reported that the distribution of the RTT 
along certain Internet paths could be modeled as a shifted 
gamma distribution. Even though the shape of the PDF in 
Figure 1 1 resembles that of a gamma function, the distribu- 
tion tails in the figure decay much slower than those of an 
exponential distribution (see below). 

Using our approach from section I V.D (i.e., assuming that 
each studied Internet random process is stationary), we ex- 
tracted the upper tails of the PDF functions in Figure 1 1 and 
plotted the results on a log-log scale in Figure 12. The figure 
shows that a straight line (without loss of generality fitted to 
the PDF of Dip in the figure) provides a good fit to the data 
(correlation 0.96) and allows us to model the upper tails of 
the PDF functions as a Pareto distribution with PDF fix) = 
afftc' 01 " 1 , where shape parameter a equals 1.16 in dataset Q x 
and 1.58 in D-^ (as before, the distribution has a finite mean, 
but an infinite variance). 


am and 6 pm EDT and shows that the increase in the delay 
during the peak hours is relatively small (i.e., by 30-40%). 
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Figure 12. Log-log plot of the upper tails of the distribution of the RTT 
(PDF). The straight line is fitted to the PDF from D^. 

C. Variation of the RTT 

We conclude the discussion of the RTT by showing that 
the round-trip delay exhibited a variation during the day simi- 
lar to that of packet loss (previously shown in Figure 5) and 
that the average RTT was positively correlated with the 
length of the end-to-end path. Figure 13 shows the average 
round-trip delay during each of eight timeslots of the day (as 
before, each point in the figure represents the average of ap- 
proximately 1,000 sessions). The figure confirms that the 
worst time for sending traffic over the Internet is between 9 
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Figure 13. Average RTT as a function of the time of day 

Figure 14 shows the average RTT sampled by the clients in 
each of the 50 U.S. states. The average round-trip delay was 
consistently high (above 1 second) for three states - Alaska, 
New Mexico, and Hawaii. On the other hand, the RTT was 
consistently low (below 600 ms) also for three states - 
Maine, New Hampshire, and Minnesota. These results (ex- 
cept Minnesota) can be directly correlated with the distance 
from New York; however, in general, we find that the geo- 
graphical distance of the access point from the East Coast had 
little correlation with the average RTT. Thus, for example, 
some states in the Midwest had small (600-800 ms) average 
round-trip delays and some states on the East Coast had large 
(800- 1 000 ms) average RTTs. 



Figure 14. Average RTT and average hop count in each of the states in 

A more substantial link can be established between the 
number of end-to-end hops and the average RTT as shown in 
Figure 14. Even though the average RTT of many states did 
not exhibit a clear dependency on the average length of the 
path, the correlation between the RTT and the number of 
hops in Figure 14 was reasonably high with p = 0.52. This 
result was intuitively expected since the RTT is essentially 
the sum of queuing and transmission delays at intermediate 
routers. 
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VII. Delay Jitter 

As we discussed above, in certain streaming situations, 
round-trip delays are much less important to real-time appli- 
cations than one-way delay jitter, because the latter can po- 
tentially cause significantly more underflow events. In addi- 
tion, due to asymmetric path conditions (i.e., uneven conges- 
tion in the upstream and downstream directions), large RTTs 
are not necessarily an indication of bad network conditions 
for a NACK-based application. For example, in many ses- 
sions with high RTTs during the experiment, the outage was 
caused by the upstream path, while the downstream path did 
not suffer from extreme one-way delay variation, and data 
(i.e., non-retransmitted) packets were arriving to the client 
throughout the entire duration of the outage. Hence, we con- 
clude that the value of the RTT was not necessarily a good 
indicator of a session's quality during the experiment and that 
one-way delay jitter should be used instead. 

Assuming that delay jitter is defined as the difference be- 
tween one-way delays of each two consecutively sent pack- 
ets, an application can sample both positive and negative val- 
ues of delay jitter. Negative values are produced by two types 
of packets those that suffered a packet compression event 
(i.e., the packets' arrival spacing was smaller than their 
transmission spacing) and those that became reordered. The 
former case is of great interest in packet-pair bandwidth esti- 
mation studies and otherwise remains relatively unimportant. 
The latter case will be studied in section VIII under packet 
reordering. On the other hand, positive values of delay jitter 
represent packet expansion events, which are responsible for 
late packets. Consequently, we analyzed the distribution of 
only positive delay jitter samples and found that although the 
highest sample was 45 seconds, 97.5% of the samples were 
under 140 ms and 99.9% under 1 second. As the above re- 
sults show, large values of delay jitter were not frequent, but 
once a packet was significantly delayed by the network, a 
substantial number of the following packets were delayed as 
well, creating a "snowball" of late packets. This fact explains 
the large number of underflow events reported in previous 
sections, even though the overall delay jitter was relatively 
low. 

vm. Packet Reordering 

A. Overview 

Real-time protocols often rely on the assumption that 
packet reordering in the Internet is a rare and an insignificant 
event for all practical purposes (e.g., [lo]). Although this 
assumptions simplifies the design of a protocol, it also make 
the protocol poorly suited for the use over the Internet. Cer- 
tainly, there are Internet paths along which reordering is ei- 
ther non-existent or extremely low. At the same time, there 
are paths mat are dominated by multipath routing effects and 
often experience reordering (e.g., Paxson[17] reported a ses- 
sion with 36% of packets arriving out of order). 


Unfortunately, there is not much data documenting 
reordering rates experienced by IP traffic over modem links. 
Using intuition, we expected reordering in our experiments to 
be extremely rare given the low bitrates of streams S, and Si. 
However, we were surprised to find out that certain paths 
experienced consistent reordering with a relatively large 
number of packets arriving our of order, although the average 
reordering rates in our experiments were substantially lower 
than those reported by Paxson [17]. 

For example, in dataset Z) lQ a , we observed that approxi- 
mately out of every three missing 14 packets one was reor- 
dered. Hence, if users of ISP, employed a streaming protocol, 
which used a gap-based detection of lost packets [10] (i.e., 
the first out-of-order packet triggers a NACK), one third of 
NACKs would be flat-out redundant and a large number of 
retransmissions would be unnecessary, causing a noticeable 
fiaction of ISP's bandwidth to be wasted. 

Since each missing packet is potentially reordered, the true 
frequency of reordering can be captured by computing the 
percentage of reordered packets relative to the total number 
of missing packets. The average reordering rate in our ex- 
periment was 6.5% of the number of missing packets, or 
0.04% of the number of sent packets. These numbers show 
that our reordering rates were at least by a factor of 10 lower 
than those reported by Paxson [ 17], whose average reordering 
rates varied between 0.3% and 2% of number of sent packets 
depending on the dataset. This diflFerence can be explained by 
the fact that our experiment was conducted at substantially 
lower end-to-end bitrates, as well as by the fact that Paxson's 
experiment involved several paths with extremely high reor- 
dering rates. 

Out of 16,852 sessions in { D Xp KjD lp ), 1,599 (9.5%) experi- 
enced at least one reordering. Interestingly, the average ses- 
sion reordering rates in our datasets were very close to those 
in Paxson's 1995 data [ 17] (12% sessions with at least one 
reordering), despite the fundamental differences in sending 
rates. The highest reordering rate per ISP in our experiment 
occurred in D\ p , where 35% of the number of missing pack- 
ets (0.2% of the number of sent packets) turned out to be re- 
ordered. In the same D\ p > almost half of the sessions (47%) 
experienced at least one reordering event. Furthermore, the 
maximum number of reordered packets in a single session 
occurred in D\ p and was 3 15 packets (7.5% of the number of 
sent packets). 

Interestingly, the reordering probability did not show any 
dependence on the time of day (i.e., the timeslot), and was 
virtually the same for all states. 

B. Reordering Delay 

To further study packet reordering, we define two metrics 
that allow us to measure the extent of packet reordering. First, 
let packet reordering delay D r be the delay from the time 
when a reordered packet was declared as missing to the time 
when the reordered packet arrived to the client. Second, let 


" Missing packets are defined as gaps in sequence numbers 
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packet reordering distance d r be the number of packets (in- 
cluding the very first out-of-sequence packet, but not the re- 
ordered packet itself) received by the client during reordering 
delay D,. These definitions are illustrated in Figure 15, where 
reordering distance d r is 2 packets and reordering delay D r is 
the delay between receiving packets 3 and 2. 


D r 


time 


Figure 15. The meaning of reordering delay D r . 

Figure 16 shows the PDF of the reordering delay D r in 
{D lp vD 2p }. The largest reordering distance d r in the com- 
bined dataset was 10 packets, and the largest reordering delay 
D r was 20 seconds (however, in the latter case, d r was only 1 
packet). Although quite large, the maximum value of D r is 
consistent with previously reported numbers (e.g., 12 seconds 
in Paxson's data [17]). The majority (90%) of samples in 
Figure 16 are below 150 ms, 97% below 300 ms, and 99% 
below 500 ms. 
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Figure 16. The PDF of reordering delay D, in {D^uD^}. 

C. Reordering Distance 

We next analyze the suitability of TCP's triple- ACK 
scheme in helping NACK-based protocols detect reordering. 
TCP's fast retransmit relies on three consecutive duplicate 
ACKs (hence the name "triple-ACK") from the receiver to 
detect packet loss and avoid unnecessary retransmissions. 
Therefore, if reordering distance d r is either 1 or 2, the triple- 
ACK scheme successfully avoids duplicate packets, and if d r 
is greater than or equal to 3, it generates a duplicate packet. 
Figure 17 shows the PDF of reordering distance d r in both 
datasets. Using the figure, we can infer that TCP's triple- 
ACK would be successful for 9 1.1% of the reordering events 
in our experiment, double-ACK for 84.6%, and quadruple- 
ACK for 95.7%. Note that Paxson's TCP-based data [17] 
show similar, but slightly better detection rates, specifically 
95.5% for triple-ACK, 86.5% for double-ACK, and 98.2% 
for quadruple-ACK. 
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Figure 17. The PDF of reordering distance d r in {Dj^uD^}. 

IX. Asymmetric paths 

Recall that during the initial executions of traceroute, our 
dialer recorded the TTL fields of each received "TTL ex- 
pired" packet. The ttl fields of these packets allowed the 
dialer to compute the number of hops between the router that 
generated a particular "ttl expired" message and the client. 
Suppose some router i was found to be located at hop in the 
upstream (i.e., forward) direction and at hop r, in the down- 
stream (i.e., reverse) direction. Hence, we can conclusively 
establish that an ri-hop end-to-end path is asymmetric, if there 
exists a router for which the number of downstream hops is 
different from the number of upstream hops (i.e., 
3z, 1 £ i < n:fi * r,). However, the opposite is not always true 
- if each router has the same number of downstream and up- 
stream hops, we cannot conclude that the path is symmetric 
(i.e., it could be asymmetric as well). Hence, we call such 
paths probably-symmetric. " 

In {D\ p \jDi p }, 72% of the sessions sent their packets over 
definitely asymmetric paths. To further understand how path 
asymmetry depends on the number of end-to-end hops, we 
extracted path information from {Z^uD^} and counted each 
end-to-end path through a particular access point exactly 
once. Figure 18 shows the percentage of asymmetric paths as 
a function of the number of end-to-end hops in the path. As 
the figure shows, almost all paths with 14 hops or more were 
asymmetric, as well as that even the shortest paths (wife only 
6 hops) were prone to asymmetry. This result can be ex- 
plained by the fact that longer paths are more likely to cross 
over AS boundaries or intra- AS adrninistrative domains. In 
both cases, "hot-potato" routing policies can cause path 
asymmetry. 

Clearly, this result depend on the particular ISP employed 
by the end-user and the autonomous systems that user traffic 
traverses. Large ISPs (such as the ones studied in this work) 
often employ numerous peering points (hundreds in our 
case), and the results shown in Figure 1 8 may not hold for 
smaller ISPs. Nevertheless, our data show that dialup home 


' 5 In the most general case, even performing a reverse traceroute from the 
server to the client could not have conclusively established each path's sym- 
metry (because the "TTL expired" messages identify router interfaces rather 
than individual routers). See [ 17] for more discussion. 
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users in general are more likely to experience an asymmetric 
end-to-end path than a symmetric one. 
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Figure 18. Percentage of asymmetric routes in {D\ p \jDi p } as a function of 
the number of end-to-end hops. 

X. CONCLUSION 

Our experiments with real-time streaming in the Internet 
indicate the following: 

• Internet packet loss is bursty, and the distribution of loss 
burst lengths and the RTT appear to be heavy-tailed; 

• one-way delay jitter appears to be much more harmful to 
low-bitrate real-time applications than either packet loss 
or large RTTs; 

• in the current Internet, RTTs in the order of several sec- 
onds are possible along paths with a slow modem link; 

• the average R1T appears to be positively correlated with 
the number of end-to-end hops, whereas packet loss does 
not semi to depend an the number of hops or the average 
RTT along the same path; 

• in the current Internet, packet reordering can be experi- 
enced even by the paths with very slow bottleneck links, 
however, both the reordering delay and the reordering 
distance are relatively small; 

• the majority of Internet paths sampled by our experiment 
were asymmetric, and a path's asymmetry could be 
linked to the path's length. 

References 

[1] A. Acharya, and J. Saltz, "A Study of Internet Round-trip De- 
lay," Technical Report CS-TR-3736, University of Maryland, 
December 1996. 

[2] M. Allman,and V. Paxson, "On Estimating End-to-End Net- 
work Parameters," ACMSIGCOMM, September 1999. 

[3] H, Balakrishnan, V.N. Padmanablah, S. Seshan, M. Stemm, 
and R.H. Katz, "TCP Behavior of a Busy Internet Server: 
Analysis and Improvements," IEEE INFOCOM, March 1998. 

[4] J. Bolliger, T. Gross, and U. Hengartner, "Bandwidth Model- 
ling for Network-Aware Applications," IEEE INFOCOM, 
March 1999. 

[5] J-C. Bolot, and T. Turletti, "Experience with Rate Control 
Mechanisms for Packet Video in the Internet" ACM Computer 
Communication Review, January 1998 (earlier paper in IEEE 
INFOCOM, 1994). 


[6] J-C. Bolot, "End-to-End Packet Delay and Loss Behavior in the 
Internet," ACMSIGCOMM, September 1993. 

[7] MS. Borella, D. Swider, S. Uludag, and G.B. Brewster, "Inter- 
net Packet Loss: Measurement and Implications for End-to-End 
QoS," International Conference on Parallel Processing, Au- 
gust 1998. 

[8] R. Caceres, P.B. Danzig, S. Jamin, D. Mitzel, "Characteristics 
of Wide-Area TCP/IP Conversations," ACM SIGCOMM, 199 1. 

[9] K. Claffy, G.C. Polyzos, and H-W. Braun, "Traffic Characteris- 
tics of the Tl NSFNET Backbone," IEEE INFOCOM, 1993. 

[10] B. Dempsey, J. Liebeherr, and A. Weaver, "On retransmission- 
based error control for continuous media traffic in packet- 
switching networks," Computer Networh and ISDN Systems, 
vol. 28, no. 5, March 1996. 

[1 1] S. Floyd, M. Handley, and J. Padhye, "Equation-Based 
Congestion Control for Unicast Applications," ACM 
SIGCOMM, September 2000. 

[12] V. Jacobson, "Congestion Avoidance and Control," ACM 
SIGCOMM, 1988. 

[ 13] D. Loguinov, and H. Radha, "On Retransmission Schemes for 
Real-time Streaming in the Internet," IEEE INFOCOM, 200 1. 

[ 14] A. Mena, and J, Heidemann, "An Empirical Study of Real Au- 
dio Traffic," IEEE INFOCOM, 2000. 

[ 15] J.C. Mogul, "Observing TCP Dynamics in Real Networks," 
ACMSIGCOMM, 1992. 

[16] A. Mukherjee, "On the Dynamics and Significance of Low- 
Frequency Components of Internet Load," Internetworking: 
Research and Experience, vol. 5, no. 4, December 1994. 

[17] V. Paxson, "Measurements and Analysis of End-to-End Inter- 
net Dynamics " Ph.D. dissertation, Computer Science Depart- 
ment, University of California at Berkeley, 1997. 

[ 18] E Radha, Y. Chen, K. Parthasarathy, and R. Cohen, "Scalable 
internet video using MPEG-4,". Signal Processing: Image 
Communication, 1999. 

[ 19] A.R. Reibman, and B.G. Haskell, "Constrains on Variable Bit- 
rate Video for ATM Networks," IEEE Transactions on Circuits 
and Systems for Video Technology, vol. 2, no. 4, December 
1992. 

[20] I. Rhee, "Error Control Techniques for Interactive Low Bitrate 
Video Transmission over the Internet," ACM SIGCOMM, Sep- 
tember 1998. 

[21] W. Tan, and A. Zakhor, "Real-Time Internet Video Using Error 
Resilient Scalable Compression and TCP-Friendly Transport 
Protocol," IEEE Transactions on Multimedia, vol. 1, no. 2, 
June 1999. 

[22] T. Turletti, and C. Huitema, "Videoconferencing Over the 
Internet," IEEE/ACM Transactions on Networking, vol. 4, no. 
3, June 1996. 

[23] UUnet Latency Statistics. httD://uunet.com/network/latencv. 

[24] M. Yajnik, S. Moon, J. Kurose, and D. Townsley, "Measure- 
ment and Modelling of the Temporal Dependence in Packet 
Loss," IEEE INFOCOM, 1999. 

[25] Y. Zhang, V. Paxson, and S. Shenker, "The Stationarity of 
Internet Path Properties: Routing, Loss, and Throughput," 
ACIRI Technical Report, May 2000, Jlttp://ww.aciri.org/ 
vern/papers.html. 


293 


